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Assessing the Impact of a Multi-Agency Project on Afghan Basic 

Education 

Abstract 

This study investigates the impact of activities done in the basic education sub-sector by a 
consortium of four non-government organizations in four provinces in Afghanistan from 
January 2004 to June 2005. Using the project objectives and components as the evaluation 
framework, and data collected through survey questionnaires, structured interviews and 
classroom observations, the study examines the impact of community participation in school 
construction and basic education planning and management, analyzes the effect of teacher 
training on teacher morale and classroom practice, and discusses lessons learned about the 
design, implementation and evaluation of the project in a context of low resources and 
limited human capacity in a post-conflict scenario. 

I. Introduction 

In 2002, four non-government organizations (NGOs) based in the United States and an 
American higher education institution formed the Afghanistan Basic Education Consortium 
(ABEC) in response to a request from the Asian Development Bank (ADB) and the Afghan 
Ministry of Education (MoE) for a program that would pilot gender-sensitive, community- 
based basic education in Afghanistan. The focus of the two-year project was to increase 
access and quality for under-served groups, particularly girls, and to test innovative 
approaches to improving education access and quality. The grant from the ADB/MoE to the 
Consortium was US$4 million. The project started in mid-2003, and had a mid-term 
evaluation completed by the end of December 2003. This impact evaluation focuses on the 
period starting from January 2004 based on the request of the four NGOs. 

This paper summarizes and discusses the evaluation results with the intention of sharing 
with others the lessons learned from the ABEC project design, implementation and 
evaluation process. It starts with an introduction to the project objectives and components, 
and a review of the purpose of the evaluation and the terms of reference for the evaluator, 
which is followed by a detailed description of the methodology used, an analysis of the 
weaknesses and limitations of the evaluation, and a discussion of what lessons can be learned 
based on the findings. 

II. The ABEC Project 

The goal of the ABEC project is to reduce poverty by identifying effective approaches for 
assuring increased access to and quality of basic education for Afghan youth, especially girls 
(ADB, 2002). The specific objectives that the project was intended to achieve include: 
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• Building capacity and strengthening partnerships between the Government, schools, 
NGOs, and communities, for participatory planning and development of basic 
education for the poor; 

• Increasing access to basic education, particularly for girls in poor communities; 

• Improving the quality of basic education for children in poor communities, 
particularly for girls and other disadvantaged groups; 

• Promoting innovative community-NGO partnerships for integrated child and youth 
development services focusing on reconciliation and development in a post-conflict 
society, and 

• Supporting policy discussions of the expansion of the pilot scheme developed under 
the Japan Fund for Poverty Reduction (JFPR) Project to a broader coverage through 
follow up ADB loans (ADB, 2002).. 

To meet these objectives, the ABEC members and MoE organized their activities around 
the following six project components: 

a) Community-based basic education planning and management; 

b) Community-govemment partnership for school reconstruction and 
construction; 

c) Provision of essential teaching and learning equipment/learning materials in 
basic education; 

d) School-based teacher training; 

e) Innovative demand-based approaches to non-formal education and training 
(targeted at youth, especially girls); 

f) Project management and monitoring with participation of the Ministry of 
Education, and impact assessment (ADB, 2002). 

Implementation of the project started in four provinces, was extended to four more 
provinces by July 2004, and was completed at the end of August 2005 (which includes a two 
month no-cost extension due to school construction delays). The provinces were selected 
because the four NGOs had already worked in those places before the ABEC project. This 
impact study only covers the four provinces that were not in the scope of the mid-term 
evaluation. 

III. Evaluation Framework 

The impact evaluation of the ABEC project was conducted by the American higher 
education institution during the period of May to June 2005. The purpose of the evaluation 
was 



a) To assess the outputs for each of the six project components; 

b) To assess the impact of selected components, specifically: community based 
basic education planning and management, school construction and 
rehabilitation, school-based teacher training, and project management, 
monitoring and impact assessment; 
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c) To discuss constraints, challenges and issues that have affected project 
implementation and achievements; and 

d) To provide a forum for stakeholders to identify and consolidate best practices and 
lessons learned. 

The following terms of reference were specified by the ABEC members for the evaluator, 
within the constraints of time and the budget. 

1 . The evaluator would review key project documents, including baseline data. 

2. The evaluator would design appropriate evaluation methods and tools such as focus 
group interviews, observation tools, etc. to measure the impact of the ABEC project 
as outlined in the components. 

3. The evaluator would visit all implementing partners before the onset of the 
evaluation. 

4. The evaluator would visit project implementation areas, as security and time permits, 
with a team of 4 Afghan researchers/data collectors, including 2 females and 2 males. 
The female researchers/data collectors would ensure the inclusion of evaluation 
activities for female beneficiaries. 

5. The team would deploy itself to reach the maximum number of beneficiaries to 
document and evaluate ABEC activities. 

Due to the a-theoretical nature of the project - with activities confused with components, 
the objectives of the project and the terms of reference were adopted as the evaluation 
framework. It is important to note, however, that regarding the first term of reference, due to 
the absence of frame of reference in the baseline data, the assessment of the quantitative 
outputs for each component/activity does not make sense. For example, given the disparate 
and treacherous local conditions that affected the respective operations and implementations 
of the partners, it was neither reasonable nor practical to compare the number of teachers 
trained and the number of books distributed by different partners. In addition, with the 
absence of a measurement of student learning outcome, the impact value of these numbers 
was weakened if not lost. Therefore, the emphasis of the evaluation was on the impact of the 
project components/activities. The following section discusses the methodology used for this 
evaluation including a participatory effort to instrument design. 

IV. Methodology 

Before the start of the evaluation, a team of four Afghan data collectors was hired 
because they could get access to many sites that would have been impossible for an ex- 
patriot. The evaluation used both primary and secondary data sources. The primary sources 
included classroom observations and surveys of nearly 350 participants, who were teachers, 
village education committee (VEC) members, students, and education ministry officials in 
pre-selected sites due to security concerns. The team of four Afghans, including two women, 
was able to collect data over a three-week period using the survey questionnaires in both Dari 
and Pashto. Unfortunately, due to security concerns, the evaluator was only able to visit one 
of the four provinces. The quarterly reports produced by the Project Coordination Unit (PCU) 
were the secondary source of data. However, the data from these reports were very difficult 




5 



to analyze without some frame of reference, as mentioned previously. This section focuses 
on the process of the evaluation instrument design, data collection and data analysis, and 
some limitations and weaknesses of the evaluation are discussed as well. 

1. Instrument design 

The evaluation instruments were designed and created through a three-day workshop, 
which was attended by the four data collectors, members from the four NGOs, including their 
Afghan national staff, as well as members from the Ministry of Education, specifically 
departments of teacher training, construction, and extra-curricular affairs. The rationale 
underlying the “participatory” effort for the instrument design was based on the following 
considerations. First, these novice data collectors had limited skills and almost no knowledge 
of the project. By engaging them especially in the process of instrument design, the evaluator 
hoped to familiarize them with the project activities and the question types in the surveys. 
Secondly, due to the time and budget constraints as well as security concerns, the ex-patriot 
evaluator could not visit in person all the pre-selected sites where the evaluator could have 
gained more insights into the implementation activities of the project partners. Besides, the 
end results of the evaluation should not only serve the interests of the donor and the evaluator 
but also that of the partners, as the project was a learning experience to all the implementing 
partners working in the field in a post-conflict context. Finally, because the partners had been 
doing their respective activities in a rather independent manner, any evaluation in a rather 
generic manner should take into account the similarities as well as their disparate 
characteristics; the evaluation process was also a chance for the partners to exchange their 
experiences and get to know their own and the others’ strengths. 

The purpose of the workshop was to create a set of evaluation tools that could be used by 
the four Afghans to collect data from as many beneficiaries as possible who were involved 
with community development, school construction, school-based teacher training, and 
capacity building. To evaluate the impact of the components/activities, the four NGOs 
proposed “effect indicators”. Regarding “community based basic education planning and 
management” and “school construction”, they wanted to find out attitudinal changes in the 
beneficiaries, such as local education officials, school administrators and teachers, 
community members and parents, in terms of greater community involvement in education 
planning, management, monitoring and school construction. With reference to “school-based 
teacher training”, their focus was on changes particularly in teachers’ classroom practice, 
such as whether teachers were using methods and activities learned in their training. In tenns 
of “project management, monitoring and impact assessment”, they were interested in the 
extent the Project Coordination Unit (PCU) had made connections and built capacity among 
the different levels of the education ministry including district office of education (DEO), 
provincial office of education (PEO) and the Ministry of Education (MoE). 

Given the extremely short time period to create the instruments and to maintain the high 
speed exchanges between the participants, all conversations at the workshop proceeded in 
Dari until it became apparent that some issues had bogged the group, when the evaluator 
would ask for translation to move the process ahead. Consequently all the questions were 
developed by the workshop participants, while the evaluator provided guidance on certain 
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question types and fonnats but generally did not veto any question that the participants felt 
was important. All initial documentation was done completely in Dari. The evaluation 
instruments include survey questionnaires for VEC members, teachers, students, principals, 
officials from district, provincial and central education offices and a classroom observation 
checklist. The detailed process used by the evaluator to facilitate the creation of the 
evaluation instruments is summarized in appendix A. 

According to the request of the four NGOs, a series of survey questionnaires were created 
to enable some consistency in questioning and reporting, to generate responses that were 
comparable across schools, districts and provinces, and to determine changes in attitudes and 
beliefs. The surveys contained both open-ended questions and questions using a five-point 
Likert scale. Each survey was conducted through face to face interviews by the four Afghan 
natives. The reason for the “compromise” between mail-in surveys and face to face in-depth 
interviews was because of the following constraints imposed on the evaluation. 

• Large scale mail-in survey was impossible and a reasonable response rate even 
without the physical and infrastructure constraints would have had a low probability. 
Reasons for this included: the lack of a reliable postal service in the country; poor 
communication infrastructure put telephone interviewing out of the question; security 
concerns limited the evaluator’s and the data-collectors’ access to some school sites; 
and both the evaluation time frame and budget were tight for a large scale survey. 

• In-depth interviews with infonnation rich questions were limited by the lack of 
trained and experienced data collectors who knew the project fairly well. Only one 
data collector met some of the minimum qualifications and was hired through the 
official search process. The rest three were recommended by people that the four 
NGOs knew. On the whole, the data collectors had limited research experience. (See 
appendix B for the selection criteria for data collectors). 

Hence, the inadequacy of human, material and financial resources and the unfavorable 
security situations might have limited the infonnation that could have been obtained from a 
broader coverage of respondents. Since the survey questions may have to sacrifice both 
extensiveness and depth, the questionnaires may not have been able to fully uncover certain 
changes in attitudes and beliefs, and some important factors that influenced the outcome of 
activities may have been left out. 

After all the instruments had been created, a pilot test was conducted at a school site 
where community organizations were functioning and where the teachers had received some 
training. Unfortunately, the evaluator himself could not attend the pilot due to security 
reasons, and the test did not go as planned. The Afghan team of data collectors arrived late 
due to car troubles and therefore missed the school session, and other complications ensued. 
Although during that short period of time the data collectors were able to ask a few questions 
from five of the surveys, it was not nearly enough for potential problems to surface and to be 
resolved. The instruments were revised the next day based on the limited pilot experience. 
Several questions were deleted and others simplified. Since the work language was Dari 
throughout the whole instrument design and pilot process, and the evaluator did not receive a 
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translated copy in English until the end of the revision period, given the time constraints, 
substantive changes were impossible before the surveys were put into actual use. 

A major merit of this process was that the participants had complete autonomy in 
deciding what questions they would raise in the surveys to get the infonnation that they 
wanted. However, a significant drawback of this process was that the evaluator did not have 
control over the content of the questions and could not detennine if they would generate 
sufficiently high yielding responses to enable the fulfillment of the terms of reference. As it 
turned out there were a number of questions that could have been improved. 

2. Data Collection 

A total of 346 surveys were held in fifteen schools, including primary, middle and high 
schools, over a three week period. The following table lists the numbers of schools/districts 
and education offices visited and the corresponding numbers of respondents by type. The 
evaluator interviewed three managers from departments of teacher training, construction and 
extra-curricular affairs, with reference to project management and capacity building. 



Number of schools/districts/education 
offices visited 


Number of respondents by type 


15 


Teachers 


91 


15 


VEC members 


53 


15 


Students 


131 


13 


Classroom observation 


46 


7 


Principals 


16 


5 


PEO/ DEO 


9 


1 


MoE 


3 



Before the data collection was completed, the ex-patriot evaluator had to leave 
Afghanistan as scheduled, and received the electronic copies of the completed surveys as 
Word documents from the data collectors afterwards, with notes translated into English. It 
took over six weeks from the time when the data collection was supposedly over to when the 
evaluator received (presumably) ah the completed surveys. It was unknown to the evaluator 
if the data collectors had submitted ah the completed surveys and impossible also to monitor 
the Dari-English translation process. Nevertheless, despite the constraints to the evaluation 
and weaknesses in the surveys themselves, the participatory process of instrument design and 
data collection with the involvement of the four NGOs as well as the Afghan national staff as 
data collectors is worth noting, especially regarding what lessons can be learned for future 
evaluation of projects with similar features. 

3. Data Analysis 

After having received electronically ah the completed surveys from the data collectors, 
who translated ah the notes into English, the evaluator printed out the Word documents and 
recorded ah the results in an Excel document including those short responses to the open- 
ended questions. It is interesting to note the similar responses to even the “open-ended” 
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questions. General response patterns by questions were summarized as percentages; only in 
one case was repeated measures statistical analysis conducted using SPSS (Version 12); 
common themes were identified from the relatively lengthy interviews with three MoE 
officials. The findings are described in the following section. 

V. Findings 

Findings are organized corresponding to the purpose of the evaluation, i.e., to assess the 
impact of the selected components: community-based approach to basic education, school 
construction and rehabilitation, teacher-training, education management capacity building, 
and to discuss constraints, challenges and issues that affected project implementation and 
achievements. 

1. Community-based basic education planning and management 

Community participation in basic education was inferred by the establishment of a 
village education committee (VEC) and the number of meetings held. In three out of four 
provinces that the data collectors visited, almost every village had an active VEC. The 
absence of VECs in one of the provinces seemed related to school construction delays. It was 
reported from both teacher and VEC respondents that most of the VECs were mainly 
involved in deciding on school locations if the schools were newly built, checking students’ 
school attendance, observing classes, distributing teaching and learning materials, 
encouraging parents to send their children to school, and solving problems for teachers as 
well as students. 

No questions in the surveys were specifically about attitudinal changes in the 
community, but most of them centered upon the functioning of the committees and its 
capacity to help in education activities. The feedback from both the teachers and VEC 
members themselves was almost entirely positive. However, there was no elaboration or 
follow up on the direct impact of VEC activities on, for instance, increased student 
enrollment or school attendance with reference to increased access to basic education as 
stated both in the primary goal and objectives of the project. The general assumption is that 
the increased awareness of important issues by VEC members on potentially controversial 
matters, such as girl enrollment in several locations, had led to positive results. Overall, such 
community-based education planning and management activities, represented by those of the 
VECs, were very strong, though it was unknown whether these activities were in the “job 
descriptions” of the VECs, and whether there were some other responsibilities yet to be 
fulfilled. 

Both the VEC members and the teachers were asked the same questions about their plans 
for their schools. It is not surprising that, in a post-conflict context where most infrastructures 
have been destroyed, both the teachers and the VEC members expressed “ school 
construction’’'’ as the top priority, although the teachers put more emphasis on teaching and 
learning facilities, such as building libraries and computer labs. In general, teachers’ 
responses showed less knowledge about VEC functions and activities, indicating that they 
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were either not participating in VEC meetings or that the infonnation generated at these 
meetings was not always disseminated to them. 

2. Community-government partnership for school construction 

Overall, all of the respondents were very happy to have a new or refurbished school in 
their community, with almost no complaints concerning the quality or design of the 
buildings. Where new schools were built, almost all communities contributed labor and/or 
collected money to pay for the land. Looking after the construction materials was also the 
responsibility of the communities. Nevertheless, one third of the teacher respondents still 
expressed the demand for more school construction ranging from a wall around the 
schoolhouse to a library, from a garden to a conference room, from an iron roof to a 
computer lab despite the fact that few schools had reliable electricity. Very few seemed to 
believe that they could possible generate the necessary funds from their own communities to 
enable such construction; instead they seemed to expect the donors to continue providing the 
materials. Nearly half of the teacher respondents wanted to expand the newly built schools to 
include more classes at higher levels or just to allow more students to attend. Unfortunately 
again there were no direct questions in the surveys about the significance of community- 
government partnership for school construction/rehabilitation regarding, for example, 
increased student (or girl) enrollment or community ownership of the schools, or their 
attitudes towards community involvement in the form of contributing money and labor. 

3. School-based teacher training 

The majority of the teacher training workshops were not actually school based except 
those done by one of the ABEC partners. Most of the teacher respondents reported that the 
teacher training staff had observed their classes at school, ranging from three to six times in 
the past 18 months. All reported that they had attended some kind of workshops, such as 
those on teaching methodology, social development and gender issues, ranging from 5 to 40 
days in length in the same time frame. One impact of these workshops is that they have 
boosted the teachers’ morale, which is indicated by their expressed confidence in their 
students’ interest in learning, and their belief that students were learning well. 

Although most teachers (87.5%) claimed that attending the workshops enabled them to 
become more “professional” in teaching and know better how to teach, it was difficult to 
completely triangulate this either with classroom observations or from student responses. The 
data collectors did observe many procedural behaviors of the teachers, such as: “ Teacher was 
present in the beginning of the class ”, “ Teacher checked students ’ attendance ”, and “ Teacher 
wrote the topic of the new lesson on the blackboard ”. Clearly, these are the behaviors 
emphasized by the training staff from the ABEC partners since the checklist was recomposed 
from those used by the partners. Overall, the limitations of the procedure-oriented classroom 
observation checklist failed to provide information on the teachers’ “professional” expertise 
relevant to promoting students’ learning. The student survey did not give space for students’ 
comments on their teachers’ teaching or their learning. 
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Almost all of the teachers and most of the principals that participated in the survey 
emphasized specifically that one of the benefits of attending the workshops was that the 
teachers knew how to make a lesson plan, although we do not know what their lesson plan 
consisted of without a sample, and how they understood the role of a lesson plan in the 
process of teaching and learning. 

4. Capacity building in project management, monitoring and impact assessment 

The four-member Project Coordination Unit (PCU) was responsible for project 
management, monitoring and assessment capacity building at the ministry level and 
coordinating activities among all the participants. Although the PCU team was actively 
engaged in building relationships with education officials in Kabul, systematic efforts to 
build capacity among the departments of the MoE was hampered by the lack of a budget for 
training purposes as well as the capacity in the PCU team itself to train others, because the 
PCU did not have enough staff trained in education disciplines. 

Findings from the field interviews with district and provincial education officers revealed 
that most district offices were unfamiliar with a project labeled “ABEC” but knew the 
implementing partners very well as a result of their considerable participation and 
involvement in ABEC activities. They were just not aware of the umbrella ABEC 
consortium. At the provincial level, there was greater awareness of ABEC, because the 
implementing partners were practically responsible for any capacity building at this level, 
and the chief education officer was involved in the project activities. This is due to the 
hierarchy of the capacity-building strategy, which involved the provincial education officer 
first and then provincial department managers who worked more directly with district 
education officers as their time and budget allowed. 

All three department managers from the MoE, who were interviewed separately and 
privately, reported significant contact with PCU members and increased involvement in 
decision-making on the project’s direction and implementation. However, their shared major 
concern was over sustainability of what the project had achieved, which revealed a dilemma 
in donor-driven capacity building in a context of low resources and limited human capacity. 
They recognized that outside support was necessary for the continuation of whatever the 
partners had established. One of the mangers noted: 

“I am very concerned about sustainability. Certainly it is good that the 
schools were built and that some teachers were trained, but I think that 
the teacher training part will not be supported after the project. Also, 
good was the creation of community support through the village 
education committees and other community organizations. This 
additional support is good but I do not know how long it will last without 
outside support.” 

Nevertheless, in the meantime, they wondered if outside/donor support might develop 
dependence in the local beneficiaries. 
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“The other point I would like to make is that sustainability is a big 
issue. I think to have a sustainable program the people have to be self- 
motivated to want to change. If the NGOs just bring schools and 
money then the people will expect that handout to continue in the 
future. They will not be motivated to make the changes themselves.” 

One way out of the dilemma might be capacity building from within with local Afghans 
learning from doing through participating in project activities. 

“We are getting to the point in the development of the country where 
Afghans should take over more responsibilities. . . . NGOs should employ 
more Afghans from the Ministry in these projects for training purposes. 

These “seconded” people could work in both the NGO and the MoE by 
splitting their time.” 

According to the three department managers interviewed, the four ABEC 
partners/NGOs did not seem to have given priority to capacity building by engaging the local 
communities in especially the initial stage of the decision-making process. Instead, the NGOs 
proceeded based on their perceived community needs and prescribed objectives and 
activities. 

(This comment refers to ‘ grants ’ in general not just for this project). “I think that 
the nature of grants forces the project to be inflexible, both in tenns of time and in 
the output. NGOs feel like they have to supply something to communities yet I am 
not sure that sufficient time was allocated to allow the communities to voice what 
they wanted. Instead the NGOs do a very short assessment and then start 
developing these areas without sufficient time to really know if the community 
wants the project.” 

This outside implementer-centered and output-driven approach might be a result of the a- 
theoretical nature of the project where activities and objective components were confused, 
and so were output and outcome. Consequently, the partners might have to sacrifice quality 
for quantity to meet their ambitious quantifiable targets, which could lead to the potential 
long-term impact being weakened, and the sustainability of whatever capacity that had been 
build being short lived, as one of the department managers pointed out: 

“It seems that the partners have spread themselves out too much and 
that the impact of this project as a model was lost. As it is, the project 
did not have high visibility. There was nothing that could be shown as 
a model; it was just pieces. I would suggest that the partners 
concentrate in one district in a province and create something that was 
sustainable. They should make sure that each school that they built had 
all of the support services that they put in place in all these other areas. 

It really should not have been a project to put out numbers but it 
should have been a project to demonstrate quality.” 
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All three managers realized that the project was ending and therefore changes to existing 
procedures and practices were not realistic, but their comments reflected some lessons that 
should be incorporated into any future grant awards. 

5. Constraints and Challenges to this project 

The ABEC project faced numerous challenges during the implementation of activities 
and operated under many constraints. 

1 . Changing security level. Probably this was the greatest challenge. In some 
provinces, such as Paktya and Ghazni, it was extremely difficult to begin and sustain 
work. These provinces are the most religiously conservative area, where the Taliban 
still operates, and many schools are often burned down. Westerners are not 
accepted. 

2. Distances and poor communication. In some areas, these logistical problems often 
impeded progress. During the winter the roads become impassible since they are not 
paved and no one clears them. Consequently, it was impossible to bring in materials 
for construction or to bring people together for meetings. The schools were located 
in distant villages and the partners only had offices with staff in the main towns, so 
they were often out of touch. Many of these areas had no cell phone coverage either. 

3. Lack of capable staff. All activities experienced problems finding qualified capable 
local Afghans as staff that could implement new ideas and maintain services. There 
were very few people who were even literate in remote areas, where school-age 
children were excluded from the formal education system up until when a school 
was constructed. 

4. Financial constraints. All the implementing partners reported spending much 
additional capital not provided in the low overhead budgeted in this grant. The 
“administration” part of the budget for each partner had been severely reduced in 
negotiations with the donor before the project began. As a result, they did not have 
the manpower built into the project to manage all of the activities that they 
undertook. 

5. Changing involvement level by the Ministry of Education. After the overthrow of the 
Taliban government, any investment in the educator sector was welcome by the 
MoE. They approved every project that came before them, and there were many. 
Nevertheless, they did not have the capacity to manage this large infusion of money 
and activities, and some problems started to emerge. For example, there were great 
inequalities in school construction, with some provinces seeing a great deal of 
school construction while others saw nothing. In addition, there was a political 
backlash against NGOs because the local people felt that they were “developing” for 
their own interest. Under these circumstances, the Afghan government implemented 
a new policy calling for more oversight of NGO activities and more direct 
involvement by the ministries in those development activities. The MoE thus 
became very active, participating in many levels of decision-making six months 
before the project was due to end. However, this shift in attitudes and involvement 
by the MoE created some problems, given that the project had already developed a 
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series of working conditions and routines, which were difficult to adjust in the 
limited remaining time left in the project. 

6. Difficulty in coordination in the consortium. Although considerable coordination 
was needed for the consortium to function as a synergistic group, each member had 
to proceed independently in its own geographically distinct areas based on its own 
objectives, which was difficult for the Project Coordination Unit to influence or 
monitor. The major reason for the lack of coordination among the partners was 
because it was so difficult to travel in Afghanistan that the members of one 
implementing partner never once visited another partner’s site to learn from their 
experiences, and neither was there a joint meeting held to discuss any lessons 
learned. Considering that the partners worked on a number of similar projects, such 
as teacher training, they could have benefited from sharing one another’s 
experiences and knowledge, and discussing some common issues. 

7. Lack of experience and personnel in monitoring construction activities. Most of the 
funds in this project were used to build or rehabilitate schools in all the four 
provinces. Building schools on-time and under budget in difficult environment 
requires considerable oversight and expertise. Two of the implementing partners had 
very little to no experience with large scale school construction, which means that 
they had no infrastructure within their organization to manage and monitor the 
complicated process of construction, from procurement of materials to the quality of 
workmanship. Consequently they had to hire an outside contractor and hope for the 
best. According to the Construction Department manager in the Ministry of 
Education, “. . . there were some problems with quality at several schools in the 

( province name). They could have used better materials. The problem there I think 
was that the subcontractor . . . did not have the manpower to manage the project well 
and therefore could not control for quality.” 

In addition to these findings, there were two other points worth noting. One is related to 
an attempt to evaluate students’ learning, and the other is about community attitude change 
towards “girls over 12 years old should go to school”. 

• Evaluation of students ’ learning 

All the teachers reported that their students were more interested in learning than in the 
past and were doing well in reading and writing. The teachers attributed this to their 
upgraded professional practice in teaching. However, based on the data collectors’ learning 
assessment of 126 out of 13 1 student participants, both the mean and the median scores were 
3.00 on a scale of 1 to 5 which means that the higher the score, the better the students 
learned. These results did not show a strong correlation between teacher-reported students’ 
learning and the data collectors’ assessment of the students’ learning, although we do not 
know the descriptions for the scores and on what basis the data collectors’ scored the 
participants. 

• Attitudinal change towards “girls over 12 years old should go to school ” 
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Although the data did not explicitly show increased girl enrollment or attendance, there 
was a significant change in attitude in the communities towards education of girls over the 
age of twelve. Data from the first baseline survey of 144 participants at the beginning of 
2004 indicated widespread disagreement among the communities on sending their twelve 
year old girls to school. About 18 months later after various activities had been implemented 
in the communities, the subsequent baseline survey data showed a statistically significant 
change in their attitude with almost every community stating that girls over twelve should go 
to school. (See appendix C for description of the statistical analysis of the results.) 

VI. Discussion/Lessons Learned 

There are many lessons to be learned from the design, implementation and evaluation of 
the ABEC project, depending on the perspectives and beliefs of those who participated in 
different stages of the project. As evaluators/education researchers, we believe that for a 
project in a post-conflict low-resource environment to have any long-term impact, 
sustainability should be built into every aspect of the whole project life cycle. 

1. There should be a strong link between goals/objectives and activities. 

The primary goal of the ABEC project is to reduce poverty by increasing access to and 
improving quality of basic education for Afghan youth, especially girls. The underlying 
assumption is that there is a correlation, if not a causal relationship, between access to 
education of high quality as the means and poverty alleviation as the end. This assumption 
may have been conceptualized by each of the partners in the stated project objectives, but 
these were certainly not fully translated into activities with their respective output and 
outcome aligned to the primary goal. Consequently a link was missing between where to go 
and what to do to get there; instead, the focus became how to implement the activity itself. 
Without a strong rationale behind what should be achieved, and constant reflection on the 
output and outcome, the impact and sustainability of the project would be inevitably 
questionable. 

2. Project design and implementation should be contextualized. 

The ABEC project objectives put great emphasis on girls’ access to basic education 
especially in rural Afghanistan. However, more than one Village Education Committee 
specifically asked if more funding could be available so that more boys could go to school. 
We do not have the contextual infonnation of the particular school districts regarding the 
boy/girl enrollment ratio to detennine if the request was reasonable, but this possible issue is 
worth investigating. In the international community, it is a long-held belief that the gender 
disparity in access to basic education is a fundamental barrier to reach the Education for All 
(EFA) goal, that is, “by 2015, children everywhere, boys and girls alike, will be able to 
complete a full course of primary schooling” (United Nations, 2005). However, it is worth 
noting that, according to a recent study by Wils, Zhao & Hartwell (2006, forthcoming) using 
national datasets from the Demographic and Health Surveys (www.measuredhs.com), the 
greatest inequality of schooling is not in gender, but in geographical locations. Children, both 
girls and boys, in poor remote rural areas, are the predominantly marginalized group. An 
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insight into this is that project design and implementation should take into account local 
concerns and characteristics for the project to achieve the optimal desired results. 

3. Projects should be more outcome-oriented. 

Numbers were prevailing in the ABEC quarterly reports. For example, according to the 
consolidated quarterly report from the Project Coordination Unit, the following overall 
achievements had been noted as of March 31, 2005. 

Since it began its work in January 2004, the Consortium has contributed 
significantly to helping the MoE achieve its objectives in 24 districts of 
eight provinces around the country. In the target provinces, it has 
established 88 Community Education Committees; provided skills 
training to 710 teachers; constructed or rehabilitated 28 schools (with 4 
additional schools to be completed by the end of the project); enrolled 
6,150 girls and 9,590 boys in these schools; provided additional teaching 
and learning materials to 88 schools with 41,691 girl and boy students, 
centers and groups. 

These numbers, although sound very seductive, yet are difficult to comprehend without a 
frame of reference. They could show that the ABEC partners were involved in conducting a 
variety of teacher training workshops, distributing textbooks and materials, and constructing 
schools. However, we cannot assume that such output is naturally linked to improved student 
learning, an outcome expressed in one of the ABEC objectives, let alone poverty reduction. 
Moreover, in a post-conflict situation, sustaining the desirable outcome is more important 
than producing numbers. As one of the education department officer commented, too many 
projects produced numbers, but very few constructed a sustainable system that could be 
assumed by Afghan personnel. Lack of attention to outcome (effect) and sustainability could 
only result in all previous efforts in vain, and a waste of time, energy and resources. 

4. An integrated rather than a federalist consortium could be more effective. 

The survey data shows an imbalance of implementation activities across the four 
provinces; in some provinces more teachers received much more training and had many more 
schools built than in others. What is not apparent from the data is the lack of coordination 
among the four non-government organizations. Their share of responsibilities as project 
implementing partners was not based on their respective and complementary abilities, 
strengths and expertise, but rather on geographical locations. Consequently, the potential of 
the organizational synergy from the multi-agency consortium was eclipsed by individualized 
and isolated activities, which in turn, lessened the overall impact of the project. A more 
integrated rather than a federalist approach might be more effective. This lesson is crucial 
given that the tension between weak teaching capacity, inadequate capital investment and the 
demand for high quality basic education needs to be addressed with synergetic endeavors. 
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5. Teacher training should be upgraded and more systematic. 

Besides school construction, teacher training is another top priority in the ABEC 
activities. Data from the teacher survey shows that most (88%) of the participants had their 
classes observed by the staff from the organizations that conducted the training workshops, 
by their principals and VEC members. However, there was no question asked about the 
purposes of the observations, what feedback and follow-up support the teachers had received 
as a result of the observations, and what the teachers expected more from possible future 
training activities. Thus there was no information regarding potential improvement or 
adjustment for future teaching training programs. Unless teacher training in Afghanistan is 
considered a one-time activity, it is essential for the infonnation and knowledge in this area 
to be built up and documented in a systematic manner, both technically and professionally, so 
that future teacher training activities do not need to reinvent the wheel but draw upon what 
has been achieved and lessons learned. Moreover, future teacher training programs should 
focus on strengthening teachers’ professional expertise rather than procedural behavior, and 
develop teachers in their capacity of learning how to learn rather than limit training content 
in demonstrating a few teaching techniques. Given the competition for limited resources, 
especially funding, teacher training in Afghanistan needs to break through the current 
technique oriented status, so that the teachers not only have a few “fish” from the training 
workshops but also learn the skills of how to fish. 

6. Documentation and dissemination of information is important for knowledge building 
and sharing. 

It is important to realize that this project was intended as a pilot, which was designed to 
test new ideas and practices in the education sector. However, the whole piloting purpose 
was lost due to the lack of documentation of the education “successes” so that the ministry 
could leam from their efforts. As it is, most officials did not realize the work that had been 
accomplished, the successful strategies that were employed, the various methods used, or the 
insights discovered from all of these activities. Originally it was the role of the American 
university partner to provide this information but the expense associated with this task was 
eliminated in the initial budget negotiations. Consequently, much of the lessons that could 
have been learned may inevitably be forgotten. 

VII. Postscript 

The ABEC project has undertaken the initial work of building community support for a 
small number of schools in Afghanistan, and has made a contribution to the improvement of 
many lives especially in remote rural areas. The response from the communities was 
overwhelming, but the people wanted more. The communities only expressed frustration at 
the pace of change and the limited amount of materials that the project could provide given 
the budget. 



The evaluation of the project serves more as a critical reflection on the lessons learned 
about the design, implementation and evaluation of a project involving multiple agencies in a 
post-conflict scenario. The joint effort for the evaluation is phenomenal with the engagement 
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of the Afghan local people, who administered the whole data collection process, the 
expatriate evaluator as a facilitator, and the project implementing partners as consultants. 

This study not only presents findings related to the fulfillment of the project objectives, but it 
draws insights into the impact of the project from different perspectives. While the evaluation 
points out pitfalls that the project designers, implementers and data collectors could have 
avoided, it also recognizes the inevitable compromises that have to be made in a context of 
low resources, limited human capacity, restricted security, and meager time and budget. 
Moreover, this study has made an attempt to contextualize some international nonns in 
respect of community participation in basic education planning and management. 
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Appendix A: Process of Creating the Evaluation Tools 

The detailed process used by the evaluator to facilitate the creation of the evaluation tools 
is summarized below. 

i. The evaluator explained to the large group the expected outputs from the 
instrument design activity. 

ii. To illustrate the possible procedure of creating an instrument, the evaluator 
selected a specific objective of the ABEC partnership along with the various 
activities that were used to fulfill the objective. 

iii. Then one of the partners explained an implementation activity that they used to 
meet this objective. 

iv. The partners summarized to the large group their implementation activities 
pertinent to the selected objectives, highlighting the salient points including the 
outputs and expected impact so that everyone would have some reference points 
to guide them. 

v. Afterwards the participants were strategically divided into small groups, each 
with a member from one of the NGOs, a data collector and a ministry official at 
a minimum. They were asked to design questions about changes in attitudes and 
behaviors. These sub-groups then rejoined into the large group to discuss their 
questions. A list was formulated and displayed on the wall using a projector as 
each group submitted one question in succession. Each question was reviewed 
for validity and reliability. The purpose of this “brainstorm” session was to 
introduce different types of questions, the necessity for clarity, and to model the 
iterative process that would be used for the creation of all the tools. The same 
process was used to introduce each new set of questions for each new activity to 
be evaluated. 

vi. After each initial “brainstorm” session for each questionnaire, a core group of 
four “writers” were then selected based on background and gender and asked to 
start the process of finalizing that particular questionnaire. All workshop groups 
gave their first list of “brainstorm” questions for the writers to consider. This 
group left the room to work independently for three hours. Their assignment 
was to bring back a final draft for further review by the large group. 

vii. Meanwhile the remaining workshop participants started on the next 
questionnaire using the same process. The evaluator would follow the same 
iterative method: give instructions concerning the objective, have one 
implementing partner summarize an activity they used to meet that objective, 
facilitate a large-group discussion, enable a large-group effort to create pertinent 
questions, select “writers” to produce a final draft, and enable a final large 
group review of the writers’ final draft. This process was used to create six of 
the seven questionnaires. 

The exception was the questionnaire that was used specifically for interviews with 
officials from the MoE, PEO and DEO, which was created almost completely by members 
from the Ministry of Extra-Curricular Affairs and the Project Coordination Unit. 
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The creation of the classroom observation checklist used an abbreviated process. Since 
all of the four NGOs were already using some form of classroom observations, the facilitator 
chose to assign four writers to assemble one master checklist incorporating elements from the 
four partners’ lists. This was possible since there was considerable overlap among the 
checklists regarding expectations from the teachers after the trainings. 
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Appendix B: Criteria for Data Collector Selection 

Right before the arrival of the evaluator, the ABEC members advertised the four 
positions as data collectors on some bulletin boards in Kabul. The minimum qualifications 
were specified as follows: 

• BA in social science. 

• At least four years of experience in research. 

• At least two years of research experience with a research company. 

• Has been involved at least once in the last two years in Education Projects research. 

• Fluent in Dari and Pashto languages, some level of English speaking is required. 

• Flexible in working hours. 

• Should be able to travel outside of working hours and in other provinces other than 
Kabul. 
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Appendix C: Statistical Analysis of Change in Attitudes towards 
Girls over 12 Years Old Attending School 

The values of -2, -1, 1 and 2 are respectively assigned to the attitude scale from Strongly 
Disagree to Strongly Agree, i.e., Strongly Disagree = -2, Disagree = -1, Agree = 1, Strongly 
Agree =2. Using SPSS, repeated measures is employed as the statistical analysis technique. 

The results reveal that 

• regarding the attitudes towards that “girls under the age of 12 should go to school”, 
there is no statistically significant difference between the baseline surveys I and II (p 
= 0.205); 

• with reference to the attitudes towards that “girls older than 12 should go to school”, 
the difference between the baseline survey I and II is statistically significant ip = 
0.003). 




